Skip to content

chore(python): bump Python runtime from 3.9 to 3.11 - #9005

Merged
klesh merged 1 commit into
apache:mainfrom
DoDiODev:pr/python-311
Aug 3, 2026
Merged

chore(python): bump Python runtime from 3.9 to 3.11#9005
klesh merged 1 commit into
apache:mainfrom
DoDiODev:pr/python-311

Conversation

@DoDiODev

@DoDiODev DoDiODev commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Bump the Python runtime used across the project from 3.9 to 3.11. This is a
pure runtime bump: no library versions are changed and the existing Pydantic v1
code keeps working on 3.11.

Changes

File Change
backend/Dockerfile server image base python:3.9-slim-bookworm → python:3.11-slim-bookworm
backend/python/uv.sh default uv venv python 3.9 → 3.11
backend/python/pydevlake/pyproject.toml python = "~3.9" → ">=3.11,<3.12"
backend/python/plugins/azuredevops/pyproject.toml python = "~3.9" → ">=3.11,<3.12"
backend/python/test/fakeplugin/pyproject.toml python = "~3.9" → ">=3.11,<3.12"
.devcontainer/devcontainer.json python feature 3.9 → 3.11
backend/python/DevelopmentSetup.md docs now reference python3.11

Explicitly out of scope

  • No library upgrades. sqlmodel, pydantic, pytest, inflect, fire,
    jsonpointer etc. stay unchanged. Pydantic v2 / SQLModel migration and the
    corresponding poetry.lock regeneration are handled in a follow-up PR to keep
    this change reviewable and independently revertable.
  • No Node/frontend changes (the devcontainer Node feature stays at 18).

Testing

Automated (on Python 3.11.15):

  • make build-pydevlake ✅ (resolves with pydantic 1.10.26, sqlmodel 0.0.8)
  • make unit-test-python ✅ (azuredevops: 8 passed / 1 skipped, pydevlake: 10 passed)

Manual:

  • Built the server image base stage from backend/Dockerfile → image reports
    Python 3.11.15, Python remote plugins build successfully.
  • Started the azuredevops remote plugin and validated its plugin-info RPC
    handshake payload (name, extension, 12 subtasks, 14 migration scripts) on 3.11.
  • Ran a real data-collection flow (Jira, git, SonarQube) end to end.

Known unrelated issue

devcontainer build currently fails to pull mcr.microsoft.com/devcontainers/go:1-1.26-bookworm
(.devcontainer/Dockerfile). This base-image reference is not touched by this PR
and fails identically on main; it is independent of the Python 3.11 change.

Rollback

Revert this single commit; it only changes version strings and has no schema or data impact.

@DoDiODev
chore(python): bump Python runtime from 3.9 to 3.11
81233aa 
Bump the Python runtime used across the project from 3.9 to 3.11:

- backend/Dockerfile: server image base python:3.9 -> python:3.11
- backend/python/uv.sh: default uv venv python 3.9 -> 3.11
- pydevlake / azuredevops / fakeplugin pyproject.toml: python
  constraint "~3.9" -> ">=3.11,<3.12"
- .devcontainer/devcontainer.json: python feature 3.9 -> 3.11
- backend/python/DevelopmentSetup.md: docs reference python3.11

This is a pure runtime bump. No library versions are changed and the
existing Pydantic v1 code keeps working on 3.11 (verified with
build-pydevlake and unit-test-python). Library upgrades (Pydantic v2,
SQLModel, ...) and the corresponding poetry.lock regeneration are done
in a follow-up PR.

Signed-off-by: DoDiODev <DoDiDev@proton.me>
@DoDiODev DoDiODev mentioned this pull request Jul 21, 2026
Draft
klesh
klesh reviewed Jul 26, 2026
View reviewed changes

@klesh klesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you bumping python runtime? Are you using any python plugin?

@DoDiODev

DoDiODev commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

@klesh We - the company for which I'm working - would like to use devlake. The precondition is that all dependencies and the used tooling is up to date. Therefore I'm working on this step by step. The reason behind this is to reduce security issues in the software used.
When I see it correctly there is only the Azure DevOps plugin using Python. In the first step this will not be used. But we have teams also using Azure.

@klesh

klesh commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

I would recommend using the Golang Az plugin instead. The Python plugins are not actively maintained, and I am considering decommissioning them altogether to reduce the attack surface.

@DoDiODev

DoDiODev commented Aug 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@klesh Thanks — that's very helpful context, and I fully agree with the direction. Let me reframe this PR, because I think its rationale holds independently of the Python plugins:

We are not using the Python plugin. As you recommend, we will go with the Golang Azure DevOps plugin (backend/plugins/azuredevops_go). So backend/python/plugins/azuredevops / pydevlake are not the driver here — and to be explicit: we don't need a migration path. We are not running the Python plugin today, so there is no existing data or config on our side to move over.

Python remains a required runtime even without the Python plugins, because of dbt:

  • backend/python/requirements.txt pins dbt-mysql==1.7.0 / dbt-postgres==1.7.19, which the Go dbt plugin shells out to.
  • devops/docker/alpine-dbt/Dockerfile installs dbt on top of alpine:3.15 (Python 3.9).
  • devops/docker/lake-builder/Dockerfile still builds against python:3.9, which is EOL (2025-10) and therefore no longer receives security fixes — exactly the attack-surface concern you raised.
  • grafana/scripts has its own requirements.txt.

So this PR is really "stop shipping an EOL Python interpreter in our images", not "invest in the Python plugin framework". If the Python plugins are decommissioned, this PR gets smaller, not obsolete — the pyproject.toml/poetry.lock hunks would simply drop out.

One question, so I can align my follow-up work with your plans: is there an issue or a rough timeline for decommissioning the Python plugin framework (backend/python/pydevlake, the remote-plugin bridge in backend/server/services/remote)? I'd rather not open dependency PRs that touch code you intend to delete. If that is the direction, I'm happy to reduce this PR to the parts that survive the removal (Docker base images + backend/python/requirements.txt for dbt), or to help with the removal itself.

Related, and probably interesting for the same "reduce attack surface" reason: dbt-mysql has had no release since 2024-04 and pins dbt-core~=1.7, freezing the whole dbt stack on the 1.7 line and capping the usable Python version. I have a change prepared that drops dbt-mysql and moves dbt-postgres to 1.11 (dbt-core 1.12). That's a user-facing breaking change, so I'd like your opinion before opening it — happy to file an issue for discussion first.

@klesh

klesh commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

@klesh Thanks — that's very helpful context, and I fully agree with the direction. Let me reframe this PR, because I think its rationale holds independently of the Python plugins:

We are not using the Python plugin. As you recommend, we will go with the Golang Azure DevOps plugin (backend/plugins/azuredevops_go). So backend/python/plugins/azuredevops / pydevlake are not the driver here — and to be explicit: we don't need a migration path. We are not running the Python plugin today, so there is no existing data or config on our side to move over.

Python remains a required runtime even without the Python plugins, because of dbt:

  • backend/python/requirements.txt pins dbt-mysql==1.7.0 / dbt-postgres==1.7.19, which the Go dbt plugin shells out to.
  • devops/docker/alpine-dbt/Dockerfile installs dbt on top of alpine:3.15 (Python 3.9).
  • devops/docker/lake-builder/Dockerfile still builds against python:3.9, which is EOL (2025-10) and therefore no longer receives security fixes — exactly the attack-surface concern you raised.
  • grafana/scripts has its own requirements.txt.

So this PR is really "stop shipping an EOL Python interpreter in our images", not "invest in the Python plugin framework". If the Python plugins are decommissioned, this PR gets smaller, not obsolete — the pyproject.toml/poetry.lock hunks would simply drop out.

One question, so I can align my follow-up work with your plans: is there an issue or a rough timeline for decommissioning the Python plugin framework (backend/python/pydevlake, the remote-plugin bridge in backend/server/services/remote)? I'd rather not open dependency PRs that touch code you intend to delete. If that is the direction, I'm happy to reduce this PR to the parts that survive the removal (Docker base images + backend/python/requirements.txt for dbt), or to help with the removal itself.

Related, and probably interesting for the same "reduce attack surface" reason: dbt-mysql has had no release since 2024-04 and pins dbt-core~=1.7, freezing the whole dbt stack on the 1.7 line and capping the usable Python version. I have a change prepared that drops dbt-mysql and moves dbt-postgres to 1.11 (dbt-core 1.12). That's a user-facing breaking change, so I'd like your opinion before opening it — happy to file an issue for discussion first.

Thanks for the detailed explanation—I really appreciate it!

You're spot on. My plan is to deprecate the problematic dbt plugin first, which is already in progress (#8970) and scheduled for removal by August 31, 2026.

Following that, I plan to deprecate the Python subsystem over the subsequent 3 months.

I got your point and I am merging the PR for the time being.

klesh
klesh approved these changes Aug 3, 2026
View reviewed changes

@klesh klesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@klesh
klesh merged commit b7e5efa into apache:main Aug 3, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@klesh klesh klesh approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@DoDiODev @klesh